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ABSTRACT 

Previous  work  has  involved  experimental  design 
interrogation  of  a  series  of  computer  models  of  human 
task  and  workload  performance  in  military  command  and 
control  (C2)  systems.  These  models,  called  Computer 
modeling  Of  Human  Operator  System  Tasks  (CoHOST) 
(Middlebrooks  et  al.,  1999),  have  shown  a  dramatic  need 
for  some  tool  to  assist  experimental  designers  in  efforts  to 
establish  effective  experiments.  Specific  needs  include 
the  collection  of  data  in  a  manner  that  minimizes  the 
amount  of  data  required  while  maximizing  the 
effectiveness  and  power  of  the  data.  A  desktop  tool  called 
Computer-Aided  Design  Reference  for  Experiments 
(CADRE)  was  developed  for  this  purpose.  The  CADRE 
tool  includes  over  850  pages  of  reference  material 
covering  25  topics  that  are  divided  into  five  major 
sections  including  an  introduction  to  experimental  design, 
supplemental  data  collection  design  and  analysis,  basic 
analysis  of  variance  (ANOVA)  designs,  advanced 
ANOVA  designs,  and  empirical  model  building.  In 
addition,  the  CADRE  tool  contains  over  200  pages 
explaining  39  examples  of  statistical  analyses  covered  in 
the  reference  material  and  is  hyperlinked  to  Version  9.1.3 
of  the  SAS  statistical  analysis  package.  The  CADRE  tool 
can  be  used  for  choosing  experimental  design  procedures 
to  interrogate  and  build  empirical  models  in  support  of 
complex  studies  such  as  future  C2  systems  computer 
simulations. 

1.  INTRODUCTION 

While  there  are  several  published  statistical  and 
experimental  design  and  analysis  textbooks  that  cover 
basic  and  advanced  experimental  design  techniques 
(Keppel  and  Wickens,  2004;  Montgomery,  2005;  Myers 
and  Montgomery,  2002;  Shadish,  Cook,  and  Campbell, 
2002;  Winer,  Brown,  and  Michels,  1991),  and  attempts 
at  automated  experimental  design  references  (Mills, 
Meyer,  Rickels,  and  Hoyland,  1982),  there  is  a  current 


need  for  a  computer-based  reference  that  utilizes  modem 
hardware  and  software  technologies.  Such  a  tool  should 
incorporate  an  integrated  subset  of  currently  accepted 
basic  and  advanced  experimental  design  approaches  that 
are  useful  in  complex  system  research  environments 
investigated  by  human  factors  and  ergonomics 
researchers. 

Previously,  procedural  guidance  and  reference  for 
experimental  design  and  statistical  analysis  was  often 
summarized  in  hard  copy  tables  in  the  covers  of 
textbooks  (Siegel  and  Castellan,  1988).  With  the  advent 
of  hyperlink  technology  that  can  be  embedded  into 
machine-independent  Postscript  Definition  File  (PDF) 
documents,  modern  computer  technology  provides  the 
technical  basis  for  the  development  of  an  interactive 
desktop  tool  to  access  a  large  reference  resource  to  assist 
researchers  in  their  quest  for  efficient  and  effective 
designs.  An  example  of  just  how  complicated  these 
experimental  designs  can  become  in  military  capability 
studies  is  briefly  described  in  the  context  of  using 
experimental  design  to  interrogate  computer  simulations. 

2.  THE  CoHOST  EXAMPLE 

Following  the  1991  Persian  Gulf  War,  the  U.S. 
Army  determined  that  its  M577  armored  command  and 
control  (C2)  vehicle  was  obsolete.  Responding  to  this 
requirement,  the  Army  Research  Laboratory  (ARL) 
developed  a  series  of  computer  simulation  models  to 
examine  human  mental  and  physical  performance 
capabilities  in  U.S.  Army  Battalion  and  Brigade  level 
tactical  operations  centers  (TOC)  resulting  from  the 
introduction  of  a  new  vehicle  with  modernized  digital 
communications  systems.  A  computer  simulation  design 
was  implemented  with  taxonomic  based  descriptors  of 
human  performance  in  the  military  C2  domain  using 
discrete  event  programming.  Using  these  models  a  series 
of  computer  runs  was  established  to  assess  the  efficiency 
of  information  flow  and  task  loading  in  the  TOC. 
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2.1.  CoHOST  Models 

A  series  of  computer  models  called  Computer 
modeling  Of  Human  Operator  System  Tasks  (CoHOST) 
(Middlebrooks  et  al.,  1999)  were  developed  to  address 
cognitive  workload  performance  in  the  new  work  domain 
fostered  by  this  new  C2  systems  environment. 

This  effort  resulted  in  the  development  of  the 
CoHOST  series  of  models.  CoHOST  was  developed  to 
answer  questions  about  whether  human  operators  could 
operate  effectively  in  a  specified  work  domain.  An 
experimental  design  was  developed  that  allowed 
evaluative  conditions  to  be  placed  on  the  simulated 
human  operators  in  the  computer  model.  Following 
these  conditions  a  series  of  computer  runs  were  made  to 
establish  test  points  for  identified  dependent  variables 
against  specified  independent  variables.  With  these  test 
points  a  set  of  polynomial  regression  equations  was 
derived  to  describe  the  performance  characteristics 
according  to  the  dependent  variables  of  the  simulated 
human  operator  in  the  work  domain.  These  regression 
equations  were  then  used  to  predict  optimum  values  for 
the  independent  variables  as  indicators  of  maximum 
human  performance  that  could  be  achieved  in  the  work 
domain. 

2.2.  Experimental  Design  Interrogation  of  CoHOST 

CoHOST  was  then  selected  for  further  study  to  allow 
an  investigation  of  how  to  optimize  the  simulated  work 
domain  to  maximize  predicted  human  performance.  This 
simulation  was  used  as  an  environment  to  generate, 
collect,  and  analyze  empirical  data.  Experimental  designs 
were  used  (Middlebrooks  and  Williges,  2002)  to  evaluate 
performance  conditions  to  be  placed  on  the  simulated 
human  operators  in  the  computer  model  by  building 
empirical  models  to  predict  human  performance. 

After  exploring  and  rejecting  several  approaches  to 
achieve  these  experimental  objectives,  an  approach  was 
developed  that  was  an  extension  of  a  basic  25"2  fractional 
factorial  design  that  included  an  additional  level  to  allow 
an  examination  of  the  quadratic  components  of  the  main 
effects.  This  design  was  further  optimized  by 
unbalancing  the  treatment  orders  to  allow  the  computer 
simulation  requirements  to  stay  within  acceptable  limits 
(Banks,  Carson,  and  Nelson,  1996)  and  still  provide 
enough  data  to  support  the  desired  analysis.  A  set  of 
polynomial  regression  equations  was  developed  with 
these  data  to  predict  performance  characteristics  of  the 
human  operator  in  the  work  domain  simulated  in  the 
model.  The  derivative  of  the  regression  equations  was 
taken  and  set  equal  to  zero  to  provide  the  optimum  values 
for  the  independent  variables  to  produce  the  maximum 
human  performance  according  to  the  dependent  variables. 


By  using  appropriate  experimental  designs  to 
investigate  what  the  decision  makers  in  the  work  team 
deemed  to  be  important,  predictions  were  made  about 
how  the  work  domain  could  be  optimized  for  the  most 
efficient  human  performance.  An  augmented  fractional- 
factorial  design  was  developed  that  allowed  an  analysis  of 
the  main  effect  for  each  independent  variable  along  with 
predictions  of  higher-order  components  that  might  exist  in 
each  main  effect.  The  development  of  this  experimental 
design  was  accomplished  only  after  extensive  review  of 
the  literature  for  suitable  approaches  and  is  a  clear 
example  of  the  need  for  some  tool  to  assist  researchers  in 
their  attempts  to  develop  effective  and  efficient  designs. 

3.  THE  CADRE  TOOL 

The  CoHOST  experiment  is  a  real  case  example  of 
how  the  availability  of  an  experimental  design  and 
analysis  tool  can  significantly  assist  researchers  in 
experimental  design  development  and  analysis  of  the 
results.  Development  of  appropriate  experimental  designs 
for  complex  analyses  can  be  overwhelming  to  complete, 
demanding  to  structure,  and  incomplete  in  their  results  if 
an  ineffective  experiment  is  followed.  This  example  of  a 
highly  complex  study  provided  the  basis  for  developing 
procedures  and  processes  that  can  ease  the  burden  of  the 
experimental  design  development  and  ensure  that  an 
effective  design  is  selected  that  can  minimize  the  data 
collection  effort  and  maximize  the  validity  of  the  data. 

In  2003  the  Army  Research  Laboratory  (ARL),  in 
collaboration  with  the  Virginia  Polytechnic  Institute  and 
State  University  (Virginia  Tech),  sponsored  the 
development  of  just  such  a  tool  which  has  been  named 
Computer-Aided  Design  Reference  for  Experiments 
(CADRE)  and  prepared  on  a  CD-ROM  (Williges,  in 
press).  This  three  year  effort  is  now  coming  to  fruition 
and  the  resulting  interactive  experimental  design  and 
statistical  analysis  reference  resource  provides  a  highly 
effective  aid  for  researchers  quickly  to  establish  efficient 
experimental  designs  for  their  work. 

3.1.  CADRE  Structure 

The  CADRE  tool  was  designed  to  be  used  as  an 
interactive  desktop  computer  tool  to  aid  the  human 
factors  and  ergonomics  researcher  in  experimental 
design  and  analysis.  It  is  delivered  on  a  CD-ROM  in  a 
PDF  format  to  facilitate  cross-platform  computer  use. 

The  human  factors  experimental  design  and  analysis 
reference  material  in  CADRE  is  a  compilation  of  two 
technical  reports  (Slater  and  Williges,  2006;  Williges, 
2006).  The  Williges  2006  report  provides  the  primary 
reference  material  on  experimental  design  in  a 
PowerPoint  presentation  with  accompanying  notes.  Over 
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850  pages  of  experimental  design  and  analysis  reference 
material  for  human  factors  researchers  is  provided  in  this 
report.  The  content  of  the  reference  material  is  presented 
in  a  numbered  outline  format  on  PowerPoint  slides  and 
descriptive  text  together  with  appropriate  reference 
citations  are  provided  for  the  content  of  each  slide.  At 
the  end  of  each  topic  presentation,  references  are 
provided  to  the  current  scientific  literature  for 
supplemental  readings  that  are  pertinent  to  the  topic 
presented.  The  companion  report  by  Slater  and  Williges 
is  an  appendix  to  Williges’  report  and  provides  over  200 
pages  in  SAS  of  program  files  (SAS,  2004),  output  files, 
and  explanations  for  39  examples  of  statistical  analyses 
covered  by  the  2006  Williges  report. 

The  CADRE  tool  has  several  features  to  facilitate 
interactive  desktop  computer  use  for  rapid  information 
location  of  reference  material.  These  features  include 
extensive  bookmarks  within  the  PDF  file,  links  between 
the  reference  material  and  the  appendix  for  each  example 
computational  problem,  and  hyperlinks  from  the  CADRE 
tool  to  the  SAS  desktop  interactive  statistical  program 
for  each  example  problem. 

3.2.  CADRE  Topics 

The  CADRE  reference  material  covers  25  topics  and 
is  divided  in  to  five  major  sections  including  introduction 
to  experimental  design,  supplemental  data  collection 
design  and  analysis,  basic  analysis  of  variance 
(ANOVA)  designs,  advanced  ANOVA  designs,  and 
empirical  model  building  as  shown  in  Table  1 . 

Topic  1  presents  the  overall  research  design  process. 
First,  the  various  stages  of  research  are  shown  in  a  flow 
diagram.  Next  six  critical  aspects  of  this  process  are 
highlighted  beginning  with  the  research  problem  through 
research  reports.  As  with  all  subsequent  topics  covered 
in  the  reference  material,  this  topic  concludes  with  a 
summary  followed  by  suggestions  for  supplemental 
readings  for  in-depth  coverage  of  the  material  covered  in 
this  topic. 

Topic  2  is  an  introduction  to  experimental  design.  It 
begins  with  an  overview  of  various  conditions  that  can 
threaten  the  validity  of  the  results  of  experiments  and 
then  offers  a  general  notation  for  designating  designs. 
Two  general  experimental  design  alternatives  are 
presented,  and  the  case  is  developed  for  always  choosing 
randomized  experimental  designs,  if  possible. 

Topic  3  summarizes  basic  statistical  concepts  that 
are  fundamental  to  experimental  design.  This  reference 
material  is  not  a  tutorial  on  basic  statistics.  Rather,  the 
material  highlights  and  summarizes  key  statistical 
concepts.  It  is  assumed  that  users  of  this  reference 
material  already  have  a  background  in  introductory 


descriptive  and  inferential  statistics.  Consequently,  this 
topic  is  designed  as  a  review  of  basic  concepts  without 
providing  detailed  descriptions  or  mathematical 
derivations.  Users  should  refer  to  a  textbook  such  as 
Walpole’s  latest  edition  (Walpole,  Myers,  Myers,  and 
Ye,  2002)  on  introductory  statistics  for  details  on  the 
concepts  summarized  in  this  topic  if  they  are  not  familiar 
with  them.  The  concepts  of  sampling  distributions,  the 
F-distribution,  and  statistical  hypothesis  testing  are 
critical  to  understanding  the  experimental  design  topics 
covered  in  this  reference. 

Table  1.  Major  Topics  in  CADRE 


Overview 

Section  1 .  Introduction  to  Experimental  Design 
Topic  1.  Research  Design  Process 
Topic  2.  Experimental  Designs 
Topic  3.  Basic  Statistical  Concepts 
Section  2.  Supplemental  Data  Collection  and  Analysis 
Topic  4..  Supplemental  Data  Collection  Methods 
Topic  5.  Analysis  of  Nominal  Scale  Data 
Topic  6.  Analysis  of  Ordinal  Scale  Data 
Topic  7.  Summary  of  Supplemental  Data 
Section  3.  Basic  Analysis  of  Variance  (ANOVA)  Design 
Topic  8.  Introduction  to  ANOVA 
Topic  9.  ANOVA  Summary  Table  Components 
Topic  10.  Between-Subjects  ANOVA  Designs 
Topic  11.  Analysis  of  Comparisons  and  Interactions 
Topic  12.  Within-Subjects  ANOVA  Designs 
Topic.  13.  Mixed-Factors  ANOVA  Designs 
Topic  14.  Summary  of  Basic  ANOVA 
Section  4.  Advanced  ANOVA  Designs 

Topic  15.  Introduction  to  Advanced  ANOVA 
Topic  16.  Hierarchical  ANOVA  Designs 
Topic  17.  Blocking  ANOVA  Designs 
Topic  18.  Fractional  Factorial  ANOVA  Designs 
Topic  19.  Analysis  of  Covariance  (ANCOVA) 

Topic  20.  Summary  of  Advanced  ANOVA 
Section  5.  Empirical  Model  Building 

Topic  21.  Introduction  to  Empirical  Models 

Topic  22.  Multiple  Regression 

Topic  23.  Central-Composite  Designs  (CCD) 

Topic  24.  Sequential  Experimentation 
Topic  25.  Summary  of  Empirical  Models 


Topic  4  deals  with  supplemental  data  collection  that 
augments  data  from  experimental  designs.  Quantitative 
supplemental  data  are  very  important  for  understanding 
and  interpreting  research  results.  Most  supplemental 
data  are  analyzed  by  nonparametric  analyses  as  opposed 
to  parametric  analyses  used  on  the  major  dependent 
variables  manipulated  in  the  experimental  design. 
Several  types  of  supplemental  data  collection  techniques 
can  be  used.  This  reference  concentrates  on  graphical 
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rating  scales  that  are  most  often  used  to  collect 
quantitative  supplemental  data  in  human  factors  research. 

Topic  5  deals  with  an  overview  of  major 
supplemental  data  analysis  alternatives  that  can  be  used 
with  nominal  scale  data.  Only  a  sample  of 
nonparametric  analyses  covering  the  most  common 
techniques  used  in  human  factors  and  ergonomics 
research  is  presented  in  this  topic.  Siegel  (Siegel  and 
Castellan,  1988)  provides  a  detailed  discussion  of  all  of 
these  techniques,  and  their  formulae  and  notation  are 
used  throughout  this  topic  for  easy  reference.  The 
nonparametric  techniques  in  this  reference  are  organized 
around  between-subjects  and  within-subjects  techniques 
to  facilitate  an  easy  choice  of  nonparametric  analysis 
procedures  for  nominal  data. 

Topic  6  deals  with  an  overview  of  four 
nonparametric  analysis  alternatives  that  can  be  used  with 
ordinal  scale  supplemental  data.  The  four  procedures 
described  in  this  reference  material  are  often  used  in 
human  factors  and  ergonomics  research.  Once  again,  the 
Siegel  and  Castellan  reference  provides  a  detailed 
discussion  of  each  of  these  techniques,  and  their 
formulae  and  notation  are  used  throughout  this  topic  for 
easy  reference.  Similar  to  the  approach  followed  in  the 
discussion  on  nominal  data  analysis,  the  presentation  of 
this  topic  is  organized  around  between-subjects  and 
within-subjects  techniques  to  facilitate  choice  of 
nonparametric  analysis  procedures  for  ordinal  data. 

Topic  7  summarizes  the  major  points  discussed  in 
Section  2  dealing  with  supplemental  data.  First, 
supplemental  data  collection  procedures  dealing  with  self 
reports,  questionnaires,  and  rating  scales  are  reviewed. 
Second,  supplemental  data  analysis  techniques  using 
common  nominal  and  ordinal  scale  nonparametric 
procedures  are  reviewed.  Third,  a  three-step  process  for 
dealing  with  supplemental  data  in  experimental  design  is 
presented. 

Topic  8  introduces  ANOVA  designs  by  discussing 
their  advantages,  basic  terms,  and  the  three  major 
categories  of  ANOVA  designs  used  in  human  factors 
research.  Next,  procedures  for  specifying  the  underlying 
statistical  model  that  describes  the  components  of  any 
ANOVA  design  are  presented.  The  introduction  ends 
with  a  discussion  and  example  for  using  ANOVA  for 
statistical  hypothesis  testing  of  the  difference  between 
two  treatment  means. 

Topic  9  provides  an  overview  of  the  computational 
aspects  of  any  between-subjects,  within-subjects,  or 
mixed-factors  ANOVA  design  used  in  human  factors 
research.  The  subsections  are  organized  around  the  five 
major  components  of  the  ANOVA  Summary  Table  used 
for  listing  the  results  of  an  ANOVA.  Rather  than  derive 


formulae  for  calculating  each  component,  computational 
procedures  and  algorithms  are  provided.  First,  each 
component  of  the  summary  table  is  discussed  separately. 
Then  conventions  for  stating  the  complete  ANOVA 
Summary  Table  are  presented  for  each  of  the  three  major 
categories  of  ANOVA  experimental  designs  used  in 
human  factors  research. 

Topic  10  covers  the  construction  and  computational 
details  of  the  first  of  the  three  major  categories  of 
ANOVA  designs  used  in  human  factors  and  ergonomics 
research.  Between-subjects  designs  are  discussed  in 
terms  of  one-factor,  two-factor,  and  n-factor  designs. 
The  procedures  for  determining  the  sum  of  squares 
computational  formulae  for  any  ANOVA  are  presented 
in  the  discussion  of  one-factor  designs.  The  concept  of 
an  interaction  is  presented  in  two-factor  designs. 
Generalizations  for  constructing  and  analyzing  any 
between-subjects  ANOVA  design  are  summarized  under 
n-factor  designs.  Computational  examples  are  provided 
for  both  a  one-way  and  a  two-way  between-subjects 
design.  A  summary  listing  of  all  the  ANOVA  procedural 
rules  and  algorithms  for  conducting  any  ANOVA 
analysis  in  human  factors  research  is  provided  at  the  end 
of  this  topic. 

Topic  11  covers  analytical  techniques  that  can  be 
used  to  isolate  the  form  or  nature  of  the  main  effects  and 
interactions  that  are  significant  in  the  overall  ANOVA. 
Basically  these  procedures  deal  with  multiple  paired 
comparisons  of  various  treatment  means.  First,  this  topic 
covers  comparisons  and  provides  examples  of  both 
planned  and  unplanned  comparison  procedures.  Second, 
special  analytical  procedures  in  addition  to  paired 
comparisons  are  covered,  and  example  computations  are 
provided  for  the  analysis  of  interactions.  Although  these 
procedures  are  demonstrated  using  between-subjects 
designs,  these  same  techniques  are  appropriate  for 
within-subjects  and  mixed-factors  designs  covered  in  the 
next  two  topics. 

Topic  12  covers  within-subjects  ANOVA  designs  in 
which  each  subject  receives  every  treatment  condition  in 
the  human  factors  experiment.  Basic  configurations  of 
one-  and  two-factor  designs  as  well  as  generalizations  to 
n-factor  repeated  measures  designs  are  covered.  The 
advantages  of  within-subjects  designs  are  summarized  as 
compared  to  between-subjects  designs.  Additional 
considerations  in  using  repeated  measures  are  discussed 
including  the  homogeneity  of  covariance  assumption, 
balancing  techniques  for  controlling  possible 
confounding  effects  of  treatment  orders,  and  the  effect  of 
differential  transfer. 

Topic  13  covers  the  basic  configuration  and 
analytical  procedures  used  in  mixed-factors  ANOVA 
designs  which  comprise  the  third  major  category  of 
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ANOVA  designs.  Mixed-factors  designs  are  composed 
of  both  between-subjects  and  within-subjects  factors. 
This  type  of  ANOVA  design  is  often  referred  to  as  split- 
plot  designs  in  the  scientific  literature.  These  designs  are 
used  quite  frequently  in  human  factors  and  ergonomic 
research  due  to  the  nature  of  the  independent  variables 
being  investigated  in  the  experiment.  Consider  a  training 
research  study  that  investigates  training  methods  and 
practice  trials.  The  researcher  must  manipulate  the 
training  condition  variable  as  a  between-subjects  variable 
because  subjects  cannot  return  to  a  beginning  level  of 
knowledge  when  provided  with  alternative  training.  On 
the  other  hand,  practice  trials  in  this  training  experiment 
must  be  manipulated  as  a  within-subjects  factor  since 
each  subject  receives  multiple  trials.  Consequently,  a 
mixed-factors  design  is  needed.  This  topic  also  provides 
a  list  of  considerations  that  the  research  should  address 
when  using  mixed-factors  designs. 

Topic  14  provides  a  brief  summary  review  and 
roadmap  of  basic  ANOVA  in  terms  of  fundamental 
considerations,  ANOVA  design  classification,  ANOVA 
generalizations,  steps  in  the  overall  ANOVA  design  and 
data  interpretation  process.  Details  on  these  issues  are 
provided  in  specific  topics  referenced. 

Topic  15  introduces  topics  in  advanced  ANOVA 
that  expand  upon  the  basic  ANOVA  procedures,  rules, 
and  algorithms  already  described  in  Section  3  of  the 
reference  material.  Two  examples  of  extending  basic 
ANOVA  (i.e.,  Quasi-F  ratios  and  Randomized  Blocks 
Designs)  are  presented  in  detail.  In  addition,  this 
introduction  provides  an  overview  of  several  special 
purpose  ANOVA  design  and  analysis  procedures  that 
satisfy  various  constraints  present  in  human  factors  and 
ergonomics  research.  Detailed  discussions  of  these 
procedures  are  presented  in  subsequent  topics. 

Topic  16  covers  hierarchical  ANOVA  designs  that 
include  nested  factors  of  interest.  This  topic  describes 
both  the  basic  layout  and  the  ANOVA  computations 
involved  in  this  class  of  experimental  designs.  These 
basic  procedures  can  be  generalized  to  higher-order 
hierarchical  designs,  and  the  use  of  hierarchical  designs 
in  human  factors  and  ergonomics  research  is  discussed. 
This  topic  ends  with  a  general  summary  of  the 
considerations  of  using  hierarchical  designs  in  human 
factors  and  ergonomics  research. 

Topic  17  deals  with  ANOVA  experimental  designs 
that  can  be  used  to  control  nuisance  variable  that  are 
confounded  with  the  factors  of  interest  in  an  experiment. 
These  confounding  factors  may  include  the  effect  of 
repeated  testing  sessions  or  experimenter  bias  resulting 
from  using  several  experimenters  in  data  collection. 
Confounding  is  controlled  through  blocking  designs 
where  blocks  represent  the  nuisance  variable.  Through 


the  use  of  modular  representation  the  exact  nature  of  the 
block  confounding  with  specific  treatment  effects  of 
interest  can  be  determined  by  the  experimenter  in  order 
to  choose  an  experimental  design  to  avoid  confounding 
blocks  with  effects  of  major  interest  in  a  factorial 
ANOVA  design.  The  use  of  blocking  is  demonstrated 
through  2k  factorial  designs  and  extended  to  pseudo¬ 
factor  blocking. 

Topic  18  deals  with  fractional  replicates  of  full 
factorial  ANOVA  designs.  When  large-scale  factorial 
designs  are  used  in  human  factors  and  ergonomics 
research,  constraints  such  as  time,  money,  and  equipment 
availability  may  make  the  complete  factorial  design 
unfeasible  and  data  collection  must  be  restricted  to  a 
portion  of  the  design.  Obviously,  some  effects  in  the  full 
factorial  design  cannot  be  evaluated  and  some  effects 
will  be  confounded  with  others  if  data  are  not  collected 
on  the  entire  design.  The  experimenter  must  select  a 
subset  of  the  complete  design  that  yields  the  most  useful 
data.  Fractional-factorial  designs  are  ANOVA  designs  in 
which  only  a  fractional  portion  of  the  complete  factorial 
designs  are  observed.  Fractional  replications  of  2k,  3k, 
and  5k  designs  can  be  constructed  through  modular 
representation.  Most  often  2k  p  fractional  replicates  are 
used  in  human  factors  research  to  avoid  subsequent 
confounding  of  partial  effects  that  are  present  in  3k  and 
5k  designs.  Alternatively,  a  subset  of  fractional-factorial 
designs  called  Latin  Square  designs  can  be  used  to 
specify  the  subset  of  treatment  conditions  to  observe 
when  the  experimenter  is  only  interested  in  testing  the 
main  effects  of  three  factors  of  interest.  Consequently, 
this  topic  describes  both  2k"p  fractional  replicates  and 
Latin  Square  designs. 

Topic  19  deals  with  an  analytical  technique  for 
reducing  the  effect  of  a  covariate  to  increase  the 
sensitivity  of  the  F-test  on  effects  of  interest  to  the 
experiment.  The  covariate  is  correlated  with  the 
dependent  variable  and  its  effect  is  removed  through 
simple  linear  regression.  Both  calculations  of  correlation 
and  simple  regression  are  described  as  the  basic 
components  of  analysis  of  covariance  (ANCOVA). 
Basic  computations  in  ANCOVA  and  subsequent 
interpretations  of  results  are  described  in  this  topic. 

Topic  20  summarizes  the  advanced  ANOVA 
techniques  described  in  the  reference  material.  This 
topic  provides  a  summary  of  design  constraints  and 
describes  a  process  for  addressing  them. 

Topic  21  provides  an  overview  of  quantitative 
models  with  an  emphasis  on  empirical  model  building 
developed  through  efficient  experimental  design. 
Empirical  models  are  descriptive  models  of  human 
behavior  based  on  results  obtained  through  one  or  more 
controlled  experiments  that  can  be  used  to  predict  human 
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performance  in  complex  systems.  These  empirical 
models  can  assist  the  human  factors  specialist  in 
conducting  design  tradeoffs  of  critical  interface 
parameters. 

Topic  22  provides  an  overview  of  multiple 
regression  procedures  used  to  generate  empirical  models 
that  involve  more  than  one  factor.  First,  multiple  linear 
regression  is  described  to  demonstrate  the  calculations 
involved  when  considering  more  than  one  factor  in  the 
empirical  model.  Both  classical  and  modern  regression 
procedures  for  selecting  the  best  multiple  regression 
equation  are  described.  Next,  polynomial  regression  is 
discussed  as  the  general  form  for  stating  empirical 
models  in  human  factors  research. 

Topic  23  describes  experimental  designs  that  can  be 
used  to  collect  data  for  building  second-order  empirical 
models.  Specifically,  this  topic  focuses  on  central- 
composite  designs  (CCD)  that  were  developed  to  explore 
response  surfaces  using  empirical  models.  The 
background,  specification,  analysis,  and  an  example  of  a 
CCD  along  with  a  comparison  to  alternative  second- 
order  experimental  designs  are  discussed  in  this  topic. 

Topic  24  incorporates  the  previous  topics  into  an 
overall  strategy  for  conducting  research  on  complex 
systems  often  addressed  in  human  factors  and 
ergonomics  research.  Due  to  the  nature  of  complex 
research,  the  experimenter  can  conduct  a  series  of  small 
independent,  but  interrelated,  experiments  rather  than 
one  large  complex  study.  This  topic  describes  general 
strategies  for  conducting  complex  experiments  and 
specifically  draws  upon  considerations  made  in  response 
surface  methodology  for  conducting  a  series  of  small, 
interrelated  studies.  A  general  paradigm  for  conducting 
sequential  research  is  presented  along  with  a  detailed 
example  of  using  this  paradigm  in  human  factors 
research  that  resulted  in  a  set  of  guidelines  for  sequential 
experimentation.  The  topic  concludes  with  a  description 
of  combining  sequential  experiments  into  a  common 
database  that  incorporates  the  results  of  several 
experiments.  This  database  can  then  be  interrogated  to 
generate  integrated  empirical  models  across  experiments. 

Finally,  Topic  25  summarizes  the  empirical  model 
building  techniques.  The  purpose  and  characteristics  of 
empirical  model  building  experiments  are  reviewed,  the 
major  components  of  empirical  models  are  listed,  and  a 
sequential  experimentation  process  for  human  factors 
research  is  provided. 

3.3.  CADRE  Bookmarks 

The  PDF  file  in  CADRE  is  heavily  bookmarked 
around  eight  expandable  major  Bookmark  tabs  shown  in 
Table  2  that  include  over  500  bookmarks  when  fully 


expanded.  The  bookmarks  were  designed  to  provide  the 
CADRE  user  with  alternative  paths  for  rapid  location  of 
information  within  the  tool. 

Table  2.  Major  Bookmark  Tabs  in  CADRE 


Bookmark  Tab  1 .  Table  of  Contents 

Bookmark  Tab  2.  Subject  Index 

Bookmark  Tab  3.  Experimental  Design  Procedures 

Bookmark  Tab  4.  Statistical  Analysis  Procedures 

Bookmark  Tab  5.  Notes  on  Computational  Examples 

Bookmark  Tab  6.  Appendix  on  SAS  Examples 

Bookmark  Tab  7.  References 

Bookmark  Tab  8.  Supplemental  Readings 


The  first  two  major  bookmark  tabs  shown  in  Table  2 
deal  with  selecting  specific  content  in  the  reference 
material.  Bookmark  Tab  1  expands  to  provide  immediate 
access  to  various  subsections  of  each  topic  in  the  Table 
of  Contents  listed  in  Table  1.  Bookmark  Tab  2  expands 
to  an  extensive  subject  index  arranged  alphabetically  for 
easy  access  to  various  topics  in  the  reference  material. 

The  next  two  major  tab  sections  shown  in  Table  2 
expand  to  bookmarks  that  provide  direct  access  to 
various  experimental  design  and  statistical  analysis 
procedures  described  in  the  reference  material. 
Experimental  design  procedures  in  Bookmark  Tab  3  deal 
with  issues  such  as  research  stages,  critical  research 
methods,  basic  and  advanced  ANOVA  design  selection, 
paradigms  and  guidelines  for  sequential  experimentation, 
supplemental  data  collection,  and  components  of 
research  reports.  The  Tab  4  Bookmark  deals  with 
statistical  analysis  procedures  including  basic  statistics, 
various  basic  and  advanced  ANOVA  computations, 
simple,  multiple  linear,  and  polynomial  regression 
analysis,  and  empirical  model  building  procedures. 

Bookmark  Tabs  5  and  6  in  Table  2  expand  to  39 
computational  examples  provided  throughout  CADRE. 
A  list  of  these  examples  is  provided  in  Table  3. 
Bookmark  Tab  5  expands  to  a  discussion  of  statistical 
computations  for  each  example  in  the  reference  material. 
Alternatively,  Bookmark  Tab  6  provides  immediate 
access  to  the  SAS  program  Appendix  (Slater  and 
Williges,  2006)  that  describes  the  SAS  (SAS,  2004) 
statistical  analysis  procedures. 

The  final  two  major  bookmark  tabs  shown  in  Table 
2  deal  with  additional  reference  material.  Bookmark  Tab 
7  is  a  single  bookmark  to  the  reference  section  that 
includes  an  alphabetical  listing  of  all  references  used  in 
the  CADRE  tool.  Finally,  Bookmark  Tab  8  can  be 
expanded  to  provide  bookmarks  to  supplemental  reading 
for  book  chapters  and  articles  in  the  scientific  literature 
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to  elaborate  upon  the  reference  material  presented  for 
each  topic  listed  in  Table  1. 

3.4.  CADRE  Links  and  Hyperlinks 

The  PDF  file  of  CADRE  also  supports  links  between 
the  computational  descriptions  of  the  examples  in  the 
reference  material  with  the  39  examples  in  the  Slater  and 
Williges  (2006)  Appendix  as  shown  in  Table  3.  The 
reference  material  describes  the  various  steps  in  the 
statistical  calculation;  whereas,  the  use  of  the  SAS 
(2004)  statistical  package  for  conducting  the  analysis  is 
described  in  the  Appendix.  Each  example  in  the 
Appendix  provides  a  problem  description,  a  SAS  input 
file,  a  SAS  output  file,  and  output  explanation.  Notes  are 
provided  throughout  the  Appendix  to  describe  tailored 
modifications  when  using  various  SAS  procedures. 

Each  of  the  39  statistical  examples  in  the  Slater  and 
Williges  (2006)  Appendix  is  also  hyperlinked  to  Version 
9.1.3  of  the  SAS  (2004)  statistical  analysis  package.  If 
the  SAS  interactive  statistical  package  is  active  on  a 
user’s  computer,  the  CADRE  tool  can  be  directly  linked 
to  the  SAS  program  file  for  any  example  covered  in  the 
Appendix.  These  programs  also  can  be  used  as  a  guide 
for  analyzing  an  experimenter’s  specific  data  set  by 
simply  modifying  the  names  and  numbers  of  independent 
variables  and  the  specific  data  set  of  the  dependent 
variables. 

4.  DISCUSSION 

CADRE  represents  one  example  of  a  computer- 
based  desktop  interactive  tool  that  can  assist  researchers 
in  conducting  complex  systems  research.  This  tool  can 
be  expanded  to  incorporate  additional  experimental 
design  procedures  and  additional  statistical  analyses 
packages  that  can  be  hyperlinked  to  the  reference 
material  in  CADRE.  This  tool  needs  to  be  widely 
available  to  the  research  community  so  that 
improvements  based  on  usage  can  be  made  to  both  the 
content  and  interactive  nature  of  CADRE. 

In  addition,  the  approach  used  in  developing  the 
CADRE  tool  can  be  extended  to  other  desktop  tools  to 
assist  the  human  factors  researcher.  CADRE  can  be 
considered  the  first  prototype  in  a  series  of  tools.  Other 
computer-based  tools  addressing  a  variety  of  research 
methods  (e.g.,  multivariate  analysis,  survey  procedures, 
and  cognitive  task  analysis),  system  design  methods 
(e.g.,  focus  groups,  design  guidelines,  specifications,  and 
requirements),  and  evaluation  methods  (e.g.,  usability 
assessment,  operator  workload  evaluation,  situation 
awareness,  and  crew  resource  management)  can  be 
added  to  this  family  of  tools  to  be  used  by  human  factors 
professionals. 


Table  3.  Hyperlinked  SAS  Examples  in  CADRE 


Example  1 .  Interval  Estimation 
Example  2.  Single-Sample  t-Test 
Example  3.  Between-Subjects  t-Test 
Example  4.  Within- Subjects  t-Test 
Example  5.  Chi-Square  Goodness  of  Fit  Test 
Example  6.  Chi-Square  Test  of  Independence 
(2x2  Contingency  Table) 

Example  7.  Chi-Square  Test  of  Independence 
(RxC  Contingency  Table) 

Example  8.  Chi-Square  Test  of  Independence 
(Two  Additive  2x2  Partitions) 

Example  9.  McNemar  Change  Test 
Example  10.  Cochran  Q  Test 
Example  11.  Kolmogorov-Smirnov  Tests 
Example  12.  Kruskal- Wallis  One-Way  ANOVA 
Example  13.  Wilcoxon  Signed  Ranks  Test 
Example  14.  Friedman  Two-Way  ANOVA 
Example  15.  One-Factor,  Between-Subjects  ANOVA 
Example  16.  Two-Factor,  Between-Subjects  ANOVA 
Example  17.  Planned  Comparisons 
Example  18.  Unplanned  Comparisons 
Example  19.  Analysis  of  Interactions 
Example  20.  One-Factor,  Within- Subjects  ANOVA 
Example  21.  Two-Factor,  Within-Subjects  ANOVA 
Example  22.  Geisser-Greenhouse  and  Huynh-Feldt 
Corrections 

Example  23.  Testing  Order  Effects  in  Balanced  Latin 
Squares 

Example  24.  Within-Subjects  and  Between-Subjects 
Design  Comparison 

Example  25.  Two-Way,  Mixed-Factors  ANOVA 
Example  26.  Complete  Hierarchical  Between-Subjects 
Design 

Example  27.  Partial  Hierarchical  Between-Subjects 
Design 

Example  28.  Simple  Blocking  of  2k  Within-Subjects 
Design 

Example  29.  Complex  Blocking  of  2k  Within-Subjects 
Design 

Example  30.  One-Half  Replicate  of  24  Between-Subjects 
Design 

Example  31.  Linear  Correlation  Coefficient 
Example  32.  Alternative  Linear  Correlations 
Example  33.  Simple  Linear  Regression 
Example  34.  Analysis  of  Covariance  (ANCOVA) 
One-Way,  Between-Subjects  Design 
Example  35.  Multiple  Linear  Regression 
Example  36.  Best  Regression  Equation 
Example  37.  Polynomial  Regression 
Example  38.  Orthogonal,  Between-Subjects, 
Central-Composite  Design 
Example  39.  Blocked,  Within-Subjects. 

Central-Composite  Design 
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CONCLUSIONS 


Research  on  Army  command  and  control  systems 
requires  knowledge  and  use  of  complex  experimental 
design  and  analysis  procedures.  The  CADRE  tool  can 
assist  in  designing  effective  experiments  that  allows  data 
collection  requiring  minimum  effort  while  maximizing 
the  effectiveness  of  results.  Sections  4  and  5  of  CADRE 
dealing  specifically  with  fractional-factorial  designs, 
polynomial  regression,  empirical  model  building,  and 
sequential  experimentation  are  particularly  appropriate  in 
investigating  emerging  C2  systems. 

The  studies  that  provided  impetus  to  spawn  the 
development  of  the  CADRE  tool  were  part  of  on-going 
efforts  to  develop  better  understandings  of  how  to 
structure  socio-technical  work  environments.  These 
conditions  are  characterized  where  teams  function  in  a 
high  stress,  time  dependent,  and  decision  oriented  work 
setting.  Here  the  goal  is  to  optimize  human  cognitive 
performance  in  the  work  domain  found  in  military  C2 
systems.  This  work  is  especially  relevant  for  studies  of 
systems  not  yet  in  existence  such  as  the  Future  Combat 
System  (FCS)  and  Objective  Force  Warrior  (OFW). 
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